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Abstract 

In this paper, we investigate the associative memory in recurrent neural 
networks, based on the model of evolving neural networks proposed by Nolfi, 
Miglino and Parisi. Experimentally developed network has highly asymmetric 
synaptic weights and dilute connections, quite different from those of the Hop- 
field model. Some results on the effect of learning efficiency on the evolution 
are also presented. 
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1 Introduction 

As a model of associative memory in terms of the recurrent-type artificial neural net- 
works, Hopfield model [f ] and its variants have been successfully developed and made 
some important results using the statistical mechanical techniques on the analogy 
with spin glass [2, 3]. Their assumption of the symmetrically and fully connected 
weights of synapses is, however, unrealistic, since this does not arise naturally in 
biological neural systems. Some researches are found allowing asymmetric synaptic 
weights or diluting the network connections [2, 3]. They do not concern the network 
structures, but most of them merely investigate the effects on the storage capacities 
under limited conditions. 

In any of above mentioned models, the physical positions where neurons exist 
or the distances between neurons are not considered at all. It must play an im- 
portant role for the formation of biological networks in the brain. There should be 
the attempt to construct more natural models based on biologically and genetically 
plausible considerations, in which asymmetric and dilute connections are just the 
result. The genetic algorithm [4] can be a key ingredient toward such directions. 
Recently evolutionary search procedures have been combined with the artificial neu- 
ral networks (so called "the evolutionary artificial neural networks") [5]. Almost all 
of the models deal with the feedforward multi-layered networks to raise the per- 
formance of the networks and in many of them the evolution of both weights and 
architectures is not integrated. 

In this paper we apply a genetic evolutionary scheme to recurrent neural networks 
and investigate associative memory. Aiming to construct a biologically inspired 
neural network, we take into account the following two points in network formation: 

• The correspondence of genotype and phenotype should not be direct. Only 
developmental rules of neurons are encoded in the genotype representation. 

• The information on the physical positions of neurons and the distances between 
neurons should be included in the model. The resulting network is a physical 
object in space. 

While the hrst point, the indirect encoding scheme, has been studied [6, 7], few 
attempts have been made to incorporate both of the above two points, except for 
the model of "evolving (growing) neural network" by Nolh, et al. [8, 9]. So we follow 
their scheme treating a neural network as a creature in a physical environment. 

The outline of our model is as follows. (I) Prepare the individuals which have the 
genotypes encoded the information for generating networks. (2) Learn the patterns 
according to an appropriate learning rule. (3) Impose the task for retrieving the 
given patterns. (4) Some individuals which make good results in the retrieval task 
are selected and inherit the genotypes to their offspring with mutation. (5) Repeat 
(2)- (4) until appropriate generation. As the generations go on, the abilities of the 
individuals will improve. 



2 Model descriptions 

A population consists of M individuals. Each individual has a genotype encoded 
the information of creating the network structure. The genotype is divided into 
N blocks corresponding to N neurons. Each block contains the instructions for 
one neuron, namely, the "physical position" of the neuron (x and y coordinates 
in the "physical space", 2 dimensional plane in this paper), the directional angle 
a for the "axonal" growth, the axonal bifurcation angle 6, the axonal bifurcation 
length d, the "presynaptic" weight b (characterize the firing intensity and neuron's 
type, excitatory or inhibitory) and the "postsynaptic" weight a (amplify or damp 
the incoming signal) (Fig. 1). Note that these are essentially local information 
peculiar to each neuron. There is no direct information as to the connections between 
neurons. All values are randomly generated at the 0-th generation. 
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Figure 1: A Genotype consisted of N blocks and parameters specified in each block. 
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Figure 2: The growing and bifurcating process of an axon. 



From each neuron in the physical space, an axon grows and bifurcates as shown in 
Fig. 2 and connects with other neurons. The connections of neurons are determined 
by whether the axon reaches the neighbourhood r of other neurons or not. For 



convenience, we introduce the connectivity matrix c 8J (z,j = 1, 



N- c-- 



0) as 



1 if the axon from the i-th neuron reaches the j-th neuron, and otherwise 
0. Then the bare synaptic weight w - at birth, before learning, is given by 



w - = aibj if Cij = 1, and w - = if c 8J = 0. So the synaptic weights are generally 
asymmetric (u>°- ^ w -). 

After the bare weights are established, they are dressed by learning for a certain 
period of time. That is, p patterns {£f = ±1}(// = l,...,p) to be stored are 
embedded into the network by an appropriate learning rule: wf^ ew = wf^ + 8wi r 
Iterated several times in one generation (learning task), the dressed weights are 
obtained. In general, the synaptic increment Swij can be modulated in each learning 
step. Then the "retrieval task" is imposed to the neural network (phenotype) with 
the dressed weights Wij. Concretely, one of p patterns with random noise is given to 
the individual network as the initial states of neurons. And the state of the neuron 
is updated synchronously according to the equation Si(t + 1) = f(J2 1= i w ij s j(t))^ 
where s 8 (t)( — 1 < s 8 - < 1) is the state of the i-th neuron and t is the time step at 
retrieving. The transfer function f(x) is usually taken to be f(x) = tanh(x/2e). 
During the update proceeds, we trace the overlap of the state of the network with 
the target pattern: m^(t) = 1/N J2i=i £i s i(t)- This procedure is applied to all p 
patterns. 

The retrieval task is repeated several times with changing the noise of the set 
of p patterns, because the proper network should have the balanced robustness 
against the noise. For this reason, we take the fitness as the sum of the values of 
overlaps over all tasks. At the end of one generation, some individuals which have 
got higher fitness values are selected. Their genotypes are inherited to their offspring 
with random mutation (we do not use crossover for simplicity). The above process 
continues for appropriate generations. 

3 Simulations and results 

To carry out the experiments, we have made the following choices of parameters. A 
population consists of 50 (M = 50) individuals. The best 10 of individuals which 
have got higher fitness values are selected, and 5 copies of each of their genotypes are 
inherited to their offspring with mutation. The mutation rate is 0.004. We take a 
square, normalized to lxl, as the physical space. The ranges of each parameter for a 
genotype are < x, y < 1, < a < 2vr, < 9 < vr/3, < d < 0.15, -2 < b < 2 and 
< a < 2. The axon bifurcates 5 times and the neighbourhood of a neuron is fixed 
to r = 0.05. And the steepness parameter e in the transfer function is set to 0.015. 
In learning and retrieval tasks, we use the 7x7 non-orthogonal 4 (p = 4) patterns 
as a set of target patterns (Fig. 3), then the total number of neurons becomes 49 
(N = 49). 

As the learning rule we here introduce simple Hebbian rule for the connected 
neurons: Sw^ = AJ^^iCfCj 1 if c ij = 1? an d ^ w ij = if c 8J = 0, where A is the 
"learning efficiency" represents the intensity of a posteriori learning after the birth. 
Note that we do not take into consideration the evolution of learning rule itself. 
In this paper we make A fixed through generations. Learning and retrieval task 
processes in one generation are described by Fig. 4. First, 15 steps are given for 
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Figure 3: An example of a set of target patterns. 



RT1 RT2 RT3 RT4 
1 1 1 



Learning Retrival Tasks 
Task 

-* 

1 Generation 



Figure 4: Flow of learning and retrieval task processes in one generation. 

learning. Secondly, for each of 4 patterns 15 steps are assigned to retrieving trial by 
starting with the noisy pattern state with 5% random noise. Then the overlaps are 
marked at the step t=5,10 and 15 and summed up. The retrieval task is repeated 4 
times successively (RT1~RT4) by changing the noise. 

Bold solid lines in Fig. 5 show the transition of the htness of the best individual 
until the 1000th generation at A = 0.002 (Fig. 5(a)) and A = 0.05 (Fig. 5(b)), re- 
spectively. The htness value is normalized to max 100. The "bare weight" line(thin) 
represent the ability of the bare weights of the best individual, made by removing the 
learned changes from the dressed weights, namely the "native ability" for retriev- 
ing. The "pure genetic" line(dashed) represents the ability of genetic algorithm only 
without learning after the birth (A = 0), namely means acquisition of the "inherited 
memory". Changing the learning efficiency A, we found that higher the parameter A 
becomes, more the native ability goes down. It appears that the presence of learning 
helps the network to fit its environment and makes evolution easier. 

Fig. 6 and Fig. 7 show the network structures of the best individual and the 
distributions of distance versus weight value between connected neurons generated at 
the 0th generation(GO) and the lOOOth(GlOOO) in the case of A = 0.05, respectively. 



Here we cite the comparison of our numerical results with those of the conven- 
tional Hopheld model. In the following, H indicates the Hopheld model and E 
indicates the best individual of our evolutionary model of A = 0.05 at the 1000th 
generation. The total numbers of connections are H:2352 and E:406, then the mean 
values of the number of connections per neuron become H:48 and E:8.29. This 
shows E develops strongly diluted connections. The values of symmetry parameter, 



defined by tj = Yl 



WaWji/ J2 t ,- wh 
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are given by H:1.0 and E:0.23. This means E 
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Figure 5: Evolution of fitness of the best individual in population, (a) A = 0.002. 
(b) A = 0.05. 
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Figure 6: Network structures of the best individual at GO and GfOOO (A=0.05). 

forms highly asymmetric weights. 

Finally we mention about the fitness values. E can reach fOO fitness value for 
noisy(5%) patterns, but H cannot reach fOO and gets 88.3 in average naively (re- 
member that we admit non-orthogonal patterns). And E is almost stable for suf- 
ficient retrieving time. This suggests that our evolutionary approach may exceed 
the Hopfield model in associative memory, in spite of quite different structure of 
networks. 



4 Summary 

We have presented the results of simulations of associative memory in the recurrent- 
type neural networks composed of growing neurons. We found that the developed 
networks are far different from the Hopfield model in respect of the symmetricity 
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Figure 7: Dressed weight value versus distance between connected neurons of the 
best individual at GO and G1000 (A=0.05). 



and the connection number per neuron of the connection weights, but it is possible 
to act as a sufficient or a better associative memory than the conventional Hopheld 
model. 

In this paper we kept our model simple, in terms of minimizing computational 
requirements and studying the feasibility of the methodology of biologically inspired 
associative memory models. 

Further simulations are going to concentrate on the influence of increasing the 
sets of task-patterns. It is expected that the dynamic environment where diverse 
tasks are present excludes the task-specificity and makes us possible to examine the 
relationship between learning and evolution [10, 11]. 

It seems interesting as a future direction of research to incorporate epigenetic fac- 
tors [12] such as the primary processes of neuronal development (adhesion, apoptosis 
etc.) and/or the action of nerve growth factor from target neurons. 
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